We first introduce some notations. In appendix D, we present some numerical experiments. Lemma 3. J (θ) is L Before we prove the main statement, we first drive some boundedness and Lipschitz properties. Choose W Uniform(0, 1,...,T 1) We note that under the i.i.d. Their proofs can be found in Appendices B.1.1 to B.1.4.

When applying a stochastic algorithm, one must choose an order to draw samples.

Then, we have that 1. y Fact 2. Let z:= [ x; y ] and z This can be easily proven using the AM-GM inequality. Fact 3. Let z:= [ x; y ] R It is a crucial building block for the algorithms in this work. The following classical theorem holds for AGD. We will start by giving a precise statement of Algorithm 1.Algorithm 1 Alternating Best Response (ABR)Require: g (,), Initial point z The basic idea is the following. The following two lemmas about the inexact APP A algorithm follow from the proof of Theorem 4.1 [ Here we provide their proofs for completeness.

In this paper, we study the decentralized composite optimization problem with a non-smooth regularization term.

V null, (15) which is the expectation of the estimated update in line 5 of Algorithm 1. A.1 Robust Bellman operator is a contraction It was shown in [Iyengar, 2005, Roy et al., 2017] that the robust Bellman operator is a contraction. Here, for completeness, we include the proof for our R-contamination uncertainty set. In this section, we develop the finite-time analysis of the Algorithm 1. B.1 Notations We first introduce some notations. D. (44) Hence from the Bernstein inequality ([Li et al., 2020]), we have that |k This hence completes the proof.Lemma 4. F or any t T, |k In this section we prove Theorem 4. First note that for any x,y R In this section we develop the finite-time analysis of the robust TDC algorithm. For the convenience of proof, we add a projection step to the algorithm, i.e., we let θ The approach in [Kaledin et al., 2020] transforms the D.1 Lipschitz Smoothness In this section, we first show that J (θ) is Lipschitz.